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Evans, et al. (2016) 


¢ Historical usage required for ATC route acceptance 
¢ Other factors also contribute to ATC acceptance 


¢ Objective: Develop a predictor of operational acceptability 
for route advisories 


Approach 


Identify features ¢ Literature review 
J ¢ Subject matter experts 


Extract data ¢ ATC accepted routes 
J ¢ ATC rejected routes 


Data Mining ¢ Binary classifier 
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Model Development 


Feature Selection ¢ Forward search through 


10 features 


¢ Logistic regression 


Model Selection ¢ Decision tree 
¢ Support Vector Machine 
(SVM) with sigmoid kernel 
j ¢ Random forest 


¢ Adaptive Boost (AdaBoost) 
Parameter Selection + e.g., Number of trees 
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Model Validation ¢ Nested cross-validation 


Feature Selection 


* Forward Search with Random Forest, 10-fold cross validation 
¢ 317 to 544 observations — 40% to 48% Rejected; 60% to 52% Accepted 
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Model Selection 


¢ 7 features 
¢ 10-fold cross validation 
¢ 317 observations — 48% Rejected; 52% Accepted 
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SVM Logistic Decision AdaBoost Random 
Regression __ Tree Forest 


¢ Parameter Selection: 40 trees 


Model Validation 


¢ 7 features, Random Forest, 40 trees 
¢ Nested 10-fold cross validation 
¢ 317 observations — 48% Rejected (positive); 52% Accepted (negative) 
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Conclusions 


¢ Developed a predictor of operational acceptability for 
route advisories: 
— Accuracy of 74% 
— Route rejection predicted at rate of 88% 


¢ Relevant model features: 
— Historical usage 
Timing/location of request in maneuver start sector 
— Number of downstream sectors 
— Direct routing or via auxiliary waypoints 
— Demand to capacity levels in maneuver start sector 


¢ Best performing model is Random Forest with 40 trees 


Future Work 


¢ Extension to other airspace 


— Trial data for NAS Constraint Evaluation and Notification Tool 
(NASCENT) 


¢ Improve features 
— Include weather impact on maneuver start sector capacity 
— Add other features, e.g., Center information 


— May use voice recordings to identify timing and details of pilot 
requests to ATC 
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40% of dispatcher accepted routes see actual savings 


McNally, D., Sheth, K., Gong, C., Sterenchuk, M., Sahlman, S., Hinton, S., Lee, C., Shih, F-T., “Dynamic Weather Routes: Two 
Years of Operational Testing at American Airlines,” Air Traffic Control Quarterly, Vol. 23, No. 1, pp. 55-81, 2015. 
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Nested Cross Validation 
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Feature Selection: Forward Search 


¢ Random Forest, 10-fold cross validation 
* 317 to 544 observations — 40% to 48% Rejected (positive); 60% to 52% Accepted (negative) 
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Feature Selection: Forward Search 


¢ Method: Random Forest, 40 trees, 10-fold cross-validation 
* Positive (Rejected or Modified) 40%; Negative (Accepted) 60% 
¢ Observations: between 317 and 544, depending on features included 


Model F-Score 


1 2 3 4 5 6 7 8 9 10 

Feature Features* Features* Features* Features* Features* Features* Features* Features* Features* 

Full Count 0.648 0.695 0.753 0.771 0.764 0.766 0.801 0.775 0.767 0.780 

Concat. Count 0.674 = = = = = 2 S 2 = 
Direct Routing 0.387 0.597 0.705 0.775 = - - - - - 
No. Sectors Over NA 0.599 0.693 0.743 0.746 0.766 0.809 0.783 0.797 - 
Max D/C Ratio 0.255 0.664 0.751 0.773 0.769 0.789 0.772 0.784 = - 
MSS Over Capacity NA 0.583 0.674 0.744 0.758 0.782 0.815 - - - 
MSS D/C Ratio 0.381 0.660 0.749 0.758 0.773 0.796 = - - - 
No. Dwnstrm. Sectors 0.484 0.667 0.755 = - - - = = = 
Time to Exit MSS 0.497 0.719 E : . . : 2 : - 
Dist. to Exit MSS 0.467 0.665 0.719 0.761 0.789 - - - - - 


* Includes feature set with highest F-Score from previous column 


¢ Feature Set with highest F-Score: 


— Concatenation Count, — Time to Exit MSS, 
— Number of Downstream Sectors, — Direct Routing 
— Distance to Exit MSS, — MSS Demand/Capacity Ratio 


— MSS Over Capacity 


Feature Selection 


¢ Forward Search, using a Random Forest and 10-fold cross-validation 
¢ Feature Set with highest F-Score: 


— Hist. Count by Segment, — Time to Exit Maneuver Start Sector, 
— Number of Downstream Sectors, — Direct Routing, 
— Distance to Exit Maneuver Start Sector, — Maneuver Start Sector Demand/Capacity Ratio, 


— Maneuver Start Sector Over Capacity. 
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Model Selection 


¢ 7 features 
¢  10-fold cross validation 
¢ 317 observations — 48% Rejected (positive); 52% Accepted (negative) 
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¢ Parameter Selection: 40 trees 
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Model Selection 


¢ 10-fold cross-validation 
¢ 317 observations — 48% Positive (Rejected or Modified); 52% Negative (Accepted) 


Logistic 


Regression Decision Tree SVM Random Forest AdaBoost 
Accuracy 0.73 0.74 0.69 0.82 0.78 
Misclassification Error O.27 0.26 0.31 0.18 0.22 
True Positive Rate/Recall O71 0.75 0.63 0.84 0.76 
True Negative Rate 0.75 0.72 0.73 0.79 0.79 
Precision 0.73 0.71 0.69 0.79 0.77 
F-score 0.72 0.73 0.66 0.82 0.77 


Receiver Operating Characteristic Curve 
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Parameter Selection: Number of Weak Learners 


¢ Random Forest, with 10-fold cross validation 
¢ 317 observations — 48% Positive (Rejected or Modified); 52% Negative (Accepted) 
¢ Parameter value with highest F-Score: 40 trees 


Receiver Operating Characteristic Curve 
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Feature Selection 


¢ Method: Forward Search, training a Random Forest with 40 trees, using 10-fold cross-validation 
¢ Metric: F-Score 

* Observations: between 317 and 544, depending on features included 

¢* Data Balancing: Positive (Rejected or Modified) 40%; Negative (Accepted) 60% 


¢ Feature Set with highest F-Score (0.815): 


— Hist. Count by Segment, — Time to Exit MSS, 
— Number of Downstream Sectors, — Direct Routing 
— Distance to Exit MSS, — MSS Demand/Capacity Ratio 


— MSS Over Capacity 
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Model Selection 


° Method: 10-fold cross-validation 
° Observations: 317 
° Data Balancing: Positive (Rejected or Modified) 48%; Negative (Accepted) 52% 


° Features: Hist. Count by Segment, Time to Exit MSS, No. Downstream. Sectors, Direct Routing, Dist. to Exit MSS, 
MSS D/C Ratio, MSS Over Cap. 


Logistic Regression Decision Tree SVM Random Forest AdaBoost 
Accuracy 0.732 0.735 0.685 0.817 0.776 
Misclassification Error 0.268 0.265 0.315 0.183 0.224 
True Positive Rate 0.711 0.750 0.632 0.842 0.763 
True Negative Rate 0.752 0.721 0.733 0.794 0.788 
Precision 0.725 0.713 0.686 0.790 0.768 
F-score 0.718 0.731 0.658 0.815 0.766 
Area Under ROC 0.818 0.767 0.770 0.886 0.864 
Average Precision 0.776 0.687 0.735 0.870 0.826 


Receiver Operating Characteristic Curve 
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Parameter Selection: Number of Weak Learners 


° Method: Random Forest, with 10-fold cross validation Metric: F-Score 
° Data Balancing: Positive (Rejected or Modified) 48%; Negative (Accepted) 52% Observations: 317 


° Features: Hist. Count By Segment, Time to Exit MSS, No. Dwnstrm. Sectors, Direct Routing, Dist. to Exit MSS, MSS 
D/C Ratio, MSS Over Cap. 


Number of Trees: 20 30 40 50 60 70 80 90 100 

Accuracy 0.798 0.801 0.817 0.798 0.798 0.795 0.808 0.792 0.785 
Misclassification Error 0.202 0.199 0.183 0.202 0.202 0.205 0.192 0.208 0.215 
True Positive Rate 0.829 0.816 0.842 0.809 0.822 0.816 0.829 0.803 0.796 
True Negative Rate 0.770 0.788 0.794 0.788 0.776 0.776 0.788 0.782 0.776 
Precision 0.768 0.780 0.790 0.778 0.772 0.770 0.783 0.772 0.766 
F-score 0.797 0.797 0.815 0.794 0.796 0.792 0.805 0.787 0.781 
Area Under ROC 0.877 0.871 0.886 0.875 0.870 0.878 0.883 0.874 0.867 
Average Precision 0.860 0.820 0.870 0.833 0.844 0.854 0.863 0.840 0.835 
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Parameter Selection: Number of Weak Learners 


¢ Method: Random Forest, with 10-fold cross validation 

¢ Metric: F-Score 

¢ Observations: 317 

¢ Data Balancing: Positive (Rejected or Modified) 48%; Negative (Accepted) 52% 


¢ Features: Hist. Count by Segment, Time to Exit MSS, No. Downstream Sectors, Direct Routing, Dist. 
to Exit MSS, MSS D/C Ratio, MSS Over Cap. 


¢ Parameter value with highest F-Score (0.815): 
— 40 trees 


Receiver Operating Characteristic Curve 
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Model Validation 


¢ Random Forest, 7 features, 40 trees 
¢ Nested 10-fold cross validation 
¢ 317 observations — 48% Rejected (positive); 52% Accepted (negative) 


Receiver Operating Characteristic Curve 
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Model Validation 


¢ Method: Random Forest, with 10-fold nested cross validation 
* Observations: 317 
* Data Balancing: Positive (Rejected or Modified) 40%; Negative (Accepted) 60% 
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Comparison to One-Class Classification 


¢ Method: 10-fold cross-validation 
¢ Observations: 317 
¢ Data balancing: Positive (Rejected or Modified) 48%; Negative (Accepted) 52% 


¢ Features: Hist. Count by Segment, Time to Exit MSS, No. Downstream Sectors, Direct Routing, Dist. to 
Exit MSS, MSS D/C Ratio, MSS Over Capacity 


Random Forest Two-Class SVM One-Class SVM 


Accuracy 0.817 0.685 0.558 
Misclassification Error 0.183 0.315 0.442 
True Positive Rate/Recall 0.842 0.632 0.211 
True Negative Rate 0.794 0.733 0.879 
Precision 0.790 0.686 0.615 


FScore 0.815 0.658 0.314 


